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EXECUTIVE  SUMMARY 


• * 
Test  'And  Evaluation  is  an  integral  and  continuing  part 

of  the  weapons  system  acquisition  cycle.  In  the  process  of 
gaining  knowledge  from  researcn  through  development  and 
production,  some  test  and  evaluation  occurs  each  step  of  the 
way.  In  the  past,  it  too  often  has  heen  compromised  as  at- 
tempts were  made  to  meet  fixed  deployment  dates  even  though 
program.'S  slipped  in  their  earlier  stages.  This  paper  investi- 
gates the  Navy's  test  and  evaluation  process  in  the  procure- 
ment Ixfe  cycle  of  ships  and  aircraft,  including  some  of  the 

i 

recent  changes  in  test  and  evaluation  concepts  brought  about 

by  the  recommendations  of  the  Blue  Ribbon  Defense  Panel  and 
' ■ \ 
DOD  Directive  ^000,1.  The  unique  problems  associated  with 

the  extremely  long  but  low  ri^sk  conventional  hull  ship  acqui- 
sition programs  ai*e  discusood  using  the  Patrol  Frigate  as  an 
example.  Included  in  the  discussion  are  1)  the  Navy's  initial 
reaction  to  the  requirements  of  DODD  500O. 1 and  their  counter- 
proposal for  a u.iique  DCP/DSARC  process  applicable  to  conven- 
tional hull  shd.ps  only,  2)  the  initial  Patrol  Frigate  test  and 
evaluation  plan  using  Land  Based  Test  Sites,  and  J>)  results 
of  DSAuC  ■*'  and  II,  In  contrast  to  conventional  hull  ship 
programs,  the  aircraft  acquisition  is  one  of  high  risk  and 
uncertainty  v/ith  a relatively  short  acquisition  cycle.  Prob- 
lems associated  witn  aircraft  testing  are  also  addressed  with 
emphasis  on  concurrency,  prototyping,  T and  E competition, 


i 


non-representative  hardware,  and  operational  test  and  evalua- 
tion. It  is  concluded  that  the  systems  integration  approach 
to  testing  may  be  the  best  answer  to  the  problems  associated 
with  Navy  test  and  evaluation.  Little  faith  is  placed  on 
policies  that  simply  shift  the  test  and  evaluation  power 
structure  or  pretend  that  concurrency  can  be  eliminated.  The 
only  realistic  hope  for  decreasing  the  concurrency  of  test 
and  evailuation  and  production  is  to  reduce  the  length  of  the 
Navy  test  and  evtiluation  process  through  improved  test  plan- 
ning» improved  efficiency,  and  the  reduction  of  redundancy  in 
testing.  It  appears  that  coordinated  and  integrated  testing 
is  the  most  likely  approach  to  result  in  significant  improve- 
ment in  Navy  test  and  evaluation  and  bears  investigation  by 
other  major  acquisition  programs  as  to  its  applicability  to 
their  test  and  evaluation  efforts. 
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SECTION  I 


INTRODUCTION 

Test  and  Evaluation  is  an  integral  and  continuing 
part  of  the  v/eapons  system  acquisition  cycle.  In  the  process 
of  gaining  ^oiowlodge  from  research  through  development  and 
production  I some  test  and  evaluation  occurs  each  step  of 
the  way.  Evaluation  of  material  prior  to  approval  for  service 
use  is  a vital  function.  In  the  past > it  too  often  has  been 
compromised  as  attempts  v/ere  made  to  meet  fixed  deployment 
dates  even  thougi:  programs  slipped  in  their  earlier  stages. 
This  study  investigates  the  Navy's  test  and  evaluation  (TScE) 
process  in  the  prccuremont  life  cycles  of  ships  and  aircraft, 
including  some  of  ’the  problems  and  recent  cuniges  in  test  and 
evaluation  concepts  brought  about  by  the  recommendations  of 
the  Blue  Ribbon  Defense  Panel  and  DOD  Directive  5000,1. 

A.  BACKGROUND 

On  1 July  1970,  the  results  of  the  Blue  Ribbon  Defense 
Panel  were  published.  The  study  encompassed  the  entire  organ- 
ization, structure  and  operations  of  the  Department  of  Defense 
including  significant  comments  about  test  and  evaluation. 

This  report  made  a clear  distinction  between  operational  test 
and  evaluation  (OT&E)  and  functional,  engineering,  or  develop- 
mental testing.  Operational  testing  is  done  to  determine  to 
what  extent  a given  system  or  material  can  meet  operational 
requirements,  "It  must  provide  advance  knowledge  as  to  v/hat 
their  capabilities  and  limitations  v/ili  bo  v^hen  they  are 
subjected  to  tlio  stresaes  of  the  onvironmont  for  which  they 


we"re  designed  (usually  combat).  Operational  testing  must  take 
into  account  the  interface  with  other  systems  and  equipment, 
tactics’,  and  techniques,  organizational  arrangements,  and  the 
human  sld.lls  and  frailities  of  the  eventual  users  (2:9).*'  ^ 
Developmental  testing  is  done  to  determine  whether  design 
and  performance  contractual  specifications  are  met.  The 
report  pointed  out  that  test  data  frequently  would  have  been 
useful  for  analyses  and  decision  making,  however,  the  needed 
data  sometimes  did  not  exist,  were  derived  from. poorly  design- 
ed tests,  or  test  conditions  did  not  permit  comparison  of 
systems.  The  report  concluded  that  OT&E  had  net,  been  adequate- 
ly managed  or  supervised  at  OSD  level,  and  that  a •higher-than- 
Service*  level  OTS;E  organization  was  needed  if  the  potential 
of  OT2;E  v/as  to  be  realized. 


I'lr.  Fitzhugh's  Blue  Ribbon  Committee  saw  great  potential 
in  a program  of  well-managed  OT&E  and  recognized  prototyping 
and  preproduction  operational  testing  as  better  alternatives 
than  weapons  systems  analysis  in  the  form  of  reams  of  paper. 

Secretai’y  Packard  issued  DepSecDef  memos  pertinent  to 
OT&E  in  February,  April,  and  August  of  1971.  The  first  direct- 
ed that  each  of  the  Services  establish  an  agency  which  is  sep- 
arate and  distinct  from  the  developing  command,  and  v;hich  re- 
ports the  results  of  its  test  and  evaluation  efforts  directly 
to  the  Chief  of  the  Service  (9:1).  In  addition  the  memo 
advised  the  establishment  of  a Deputy  Director  for  Test  and 

^ Tills  notation  will  be  used  throughout  the  report  for  sources 
of  que^’ations  and  major  references.  The  first  nuraber  is  the 
source  listed  in  the  bibliography.  The  second  number  is  tno 
page  in  the  r of ere nee. 
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Evaluation  (DDT8<E)  v/ithin  DDR&E,  , The  April  memo  presented  the 
requirements  for  the  flow  of  T£<E  information  in  terms  of  pro- 
gram milestones*  The  August  memo  to  the  Service  secretaries 
re-emphasized  that  operational  test  and  evaluation  v;ill  be 
accomplished  prior  to  the  decision  to  go  into  full  production. 
The  memo  also  stated  ''tliis  initial  operational  test  and  eval- 
uation vrtll  be  accomplished  with  operational  personnel  in  a>s 
realistic  an  operating  environment  as  possible  and  v/here 
practical » v/ill  use  pilot  or  early  production  items(7:1 )." 

B.  POD  DIRECTIVE  5000.1 

On  13  duly  1971  bod  Directive  5000,1  was  published.  This 
directive  further  clarified  the  program  decision  m^lclng  pro- 
cess^ and  emphasized  that  anyone  involved  vdth  I?DT?<E  must  sc- 

/ 

I /'  quire  an  understanding  of  DSARG  concepts.  The  T&E  effort  was 

discusspd  as  follows:  “Test  and  ^Valuation  shall  commence 

as  early  as  possible,  A determination  of  operational  suit- 
ability, including  logistic  support  requirements,  \7lll  be 
made  prior  to  large-scale  prodixction  commitments,  making  use 
of  the  most  realistic  test  envi-.ronment  possible  and  the  best 
representation  of  the  future  operational  system  available. 

The  results  of  this  operational  testing  will  be  evaluated 
and  presented  to  the  BSA.RC  at  the  time  of  the  production 

decision(i):  5)  • " 

As  a result  of  this  directive  the  Secretary  of  Defense 
(SecDef)  makes  the  decision  which  Initiates  program  commit- 
ments or  increases  those  commitments.  Currently,  the  SecDof 
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v/ith  assistance  from  the  Defense  Systemj  Acquisj.tion  Review 
Council  (DSARC)  makes  three  key  system  decisions  by  choosing 
among  alternatives  posed  in  the  Development  Concept  Paper 
(DCP),  The  specification  of  the  three  distinct  stages  in 
the  system  acquisition  process  with  DOD  level  review  between 
stages  was  designed  to  mini.rnize  concurrency  and  commitment 
of  full-scale  development  and  production  before  adequate 
information  was  available  and  analyzed,  A flow  chart  of  the 
DCP/DSARC  system  qs  outlined  in  the  directive  is  presented 
in  figure  1 , 


Each  Service  has  been  finding  unique  solutions  to’  the 
organizational  requirements,  but  the  common  denominators  are’: 
l)  A direct  link  bcjtween  those  reporting  test  results  and  their 
Service  Chief;  2)  An  early  involvement  in  the  acquisition 
process  cf  the  command  which  will  use  the  weapon  system.  In 
most  cases  the  Services  are  making  use  of  the  organizations 
already  in  oxistencG  rather  than  .opening  now  offices  with  nov/ 
responsibilities.  For  instance  Commander  Operational  Test 
and  Evaluation  Force  ( OOMOPTEVFOR)  is  becoming  a more  signi- 
•flcant  factor  in  Navy  testing  since  its  commander  has  direct 
access  to  (iNO  for  OTM-I  matters. 


C.  SECNAV  IKSTPUCT.TON  ■yOOO.l 

In  response  to  DOD  Directive  5000. 1 , the  Secretary  6f 
the  Navy  ia.su.ei  SECNAVINST  ^3000.1  for  irnnlementing  the  new 
provisions.  Due  to  tlie  vide  variety  of  naval  weapons,  the 

s 

In.struction  allov/s  v.arying  .'inr're.-chQS  to  the  conduct  of  test 
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and  evaluation*  However,  such  effort  shall  be  tailored  to 
the  needs  and  cliaraateristics  of  each  individual  acquisition 
with  prime  consideration  being  given  to  adequate  operationally 
oriented  testing.  Normally,  the  following  general  sequence 
of  events  should  prevail:  1)  laboratory/contractor  prelim- 

inary test  and  evaluation  of  breadboard  demonstration  hard-  ■ 
v;are  during  the  conceptual  effort,  2)  contract  or /development 
activity  test  and  evaluation  of  subsystems  and/or  full-scale 
prototype  during  full  scale  development,  3)  technical  test 
and  evaluation  conducted  by  the  contractor  with  Navy  partici- 
pation during  pre-production/production,  k)  initial  operational 
test  and  evaluation  (I0T8<E)  by  or  with  the  active  participation 
of ' Navy  operational  forces  prior  to  the  major  production  de- 
cision, 3)  Navy  OT&B  prior  to  approval  for  service  use  and 
inventory  acoeptance( 1 6: 13)« 


SECTION  II 
SHIP  ACQUISITION 


Although  the  approach  to  'test  and  evaluation  in  SECNAVINST 
5000.1  appears  conceptually  good,  production  of  a lov/  risk 
conventional  hull  ship  might  have  to  be  delayed  one  to  two  years 
to  allow  completion  of  the  operational  testing.  For  example, 
in  the  Patrol  Frigate  (PF)  program  the  delay  to  the  program  of 
waiting  until  the  lead  ship  was  built  and  operationally  tested 
to  start  construction  of  the  follow-on  ships  was  initially  est- 
imated to  be  about  15  months.  Delays  of  this  duration  could 

i • 

drive  the  already  undesirable , but  acceptable  dip  in  force  levels 
farther  tov/ard  an  unacceptable  force  level  shortfall.  Can  de- 
lays of  this  magnitude  and  other  resulting  consequences  be  ac- 
cepted in  major  ship  acquisitions? 

A.  DGP/DSAPC  CONSIDERATIONS 

Several  Admirals  and  Program  Managers  in  OPNAV  and 
SHIPSYSCOM  objected  to  the  use  of  the  tlu'oe  step  DCP/DSARC 
procedures  for  conventional  ship  construction  programs.  They 
felt  that  the  current,  DCP/DSAPC  .was  structured  to  provide 
proper  management  for  major.  Research  and  Development  programs 
and  that  the  highly  controlled  and  definitive  system  was,  in 
many  v;.ays,  inappropriate  for  the  management  of  a conventional 
ship  acquisition  program.  They  said  the  rigid  application  of 
this  procedure  to  conventional  displacement  hull  ship  programs 
would  result  in  unnecessary  expenditure  of  time  by  project 
personnel,  increased  expense,  and  delays  in  delivery  of  urgently 
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needed  ships  to  the  fleet.  It  v^as 'recommended  that  the  first 
essential  step  required  to  alleviate  this  problem  was  to  sep- 
arate non-R8<D  programs,  such. as  shipbuilding,  from  the  R&D 
programs  that  do  require  the  "fine  grained"  management  reviev/ 
and  test  and  evaluation  now  being  proposed  for  all  programs. 

A flow  chart  of  a proposed  modification  to  the  DCP/DSARC  pro- 
cess applicable  to  conventional  displacement  hull  shipbuilding 
programs  is  iiresented  in  figure  2, 

The  Chief  of  Naval  Operations  (CNO)  after  studying  the 
problem  believed  that  a streca.mline  of  the  decision  processes 
used  in  ship  acquisitions  was  in  order.  In  his  memorandum  to 
the  Secretary  of  the  Navy  on  10  June  1972,  the  CNO  proposed  a 
policy  for  displacement  hull  ship  programs  that  included:  1) 

one  DSARC  per  ship  progreun  to  bo  conducted  after  completion  of 
formulation  and  ship  system  design,  but  prior  to  contracting 
for  detailed  design  anvi  production,  2)  demonstration  of  ade- 
quate lOT&ii)  performance  for  sub-systems  prior  to  lai'ge  scale 
production,  3)  the  use  of  shore  based  test  sites  for  critical 
ship  system  integration  devaonstration , and  4)  elimination  of 
additional  DOD  formal  reviev.'s  if  all  significant  milestones 
are  satisfactorily  mot,  including  those  for  IOT&IO(3:2), 

The  Patrol  Frigate  program  appeals  to  be  the  first  test 
case  for  gaining  approval  of  the  Navy's  position  regarding 
DCP/D5ARC  procedures  and  ship  T&E  for  conventional  displace- 
ment hull  ship  programs,  'The  decisions  on  the  PF  T&E  will 
no  doubt  be  used  as  a precedent  in  other  simileir  programs. 


Proposed 
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B.  INITIAL  PATROL  FRIGATE  TEST  AHD  EVAI.UATION  PLAN 

Supplementing  and  anticipating  the  lead  ship  construction, 
two  individual,  full-scale  leoid-based  test  sites  (LETS)  were  to 
be  erected  for  the  propulsion  and  combat  systems  respectfully* 

In  addition  to  validating  the  Si. Ip  engineering  aspects  of 
installation  and  integration  of  the  critical  Patrol  Frigate 
systems,  the  two  land-based  test  sites  have  provided  the  fac- 
ilities to  assist  in  the  configuration  management  of  the  Patrol 
Frigate  propulsion  and  combat  prototype  systems.  In  concept 
throughout  the  life  of  the  PF  program,  these  sites  would  be 
used  to  evaluate  change  proposals  prior  to  application  to  the 
ships.  The  sites  would  be  controlled  to  insure  that  the  LBTSs 
a;  a realistic  prototype  of  the  PF  combat  and  propulsion  system. 
After  the  initial  valldat'on  of  system  integration,  the  two 
land-based  test  sites  would  also  be  used  to  validate  operation, 
maintenance  and  support  concepts  proposed  for  the  PF(U|.53“5)« 

C,  LAND-BASED  TBSTHIG 

The  central  relationship  of  these  test  sites  to  the  ship 
acquisition  schedule  is  presented  in  figure  Land-based 
testing  is  to  be  used  in  concert  with  lOT&E  plans  for  individ- 
ual equipments  not  now  in  inventory.  This  should  allow  achieve- 
ment of  the  requisite  level  of  confidence  in  ship  and  equipment 
engineering  before  a commitment  is  made  to  produce  either  in 
quantity.  The  land-based  testing  and  equipment  IOT£<E  schedules 
provide  for  ijroofing  of  key  systems  beginning  two  years  before 
completion  of  the  lead  ship,  'Diis  coincides 


with  the  planned 
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award  date  for  follov/-on  ship  construction  contracts. 

The  Patrol  Frigate  combat  system  Lsoaa-Based  .Test  Site, 
should  be  a genuine  prototype  of  the  PF  combat  system.  Its 
creation  early  in  the  PF  development  process  should  provide 
the  opportunity  and  the  mechanism  to  validate  and  document 
not  only  the  combat  system  engineering  process  but  also  the 
installation,  test  and  checkout,  and  operation  and  maintenance 
procedures  prior  to  the  delivery  of  the  first  PF.  To  achieve 
these  objectives  the  LBTSs  will  aggregate  three  essential  func- 
tions : 

1,  A functional  Checkout  Facility  to  validate  the  system 
engineering  and  electrical  compatibility  of  the  equipment, 

2,  A Computer  Program  Checkout  Facility  to  debug  the 
combat  system  comp^uter  programs, 

3,  A Physical  Mockup  to  validate  the  design  and  to  be 
the  ultimate  recipient  of  the  results  of  the  above  efforts. 

The  Physical  Mockup  v/ill  be  the  combat  system  prototype. 

Throughout  the  2ife  of  the  PF  program  the  LBTSs  should 
be  used  as  an  integral  part  of  the  .PF  configuration  management 
program.  The  Ship  Acquisition  Program  Manager  (SHAPM)  should 
require  that  all  change  proposals  be  evaluated  at  the  appropri- 
ate LETS  before  they  are  considered  by  his  change  control  board. 
Once  engineering  solutions  have  been  validated  at  the  LBTS  the 
configurations  will  be  frozen  and  controlled.  This  LBTS  approach 
allov/B  f >r  a DSAPC  dociBion  on  tlie  basis  of  the  initial  assess- 
ment of  the  PF's  operational  capabilities  before  at-sea  trj.als 


and. Will  Identify  areas  where  rigorous  atjsea  evaluation  is 
required*  . . 

According  to  the  initial  pF  test  plan  it  is  apparent 
that  the  lOT&E  would  he  completed  and  the  results  available 
only  after  all  contracts  were  finalized  and  fabrication  has 
begun  on  about  half  of  the  50  ships  in  the  contract(l5J2). 

It  is  important  to  note  that  when  construction  begins  simul- 
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taneously  at  the  three  shipyards,  the  results  of  I0T8<E  on  the 
lead  ship  as  a unit  .are  not  available  and  would  not  become 
available  for  two  years.  Should  the  members  of  the  hSARC 
consider  themselves  obligated  to  be  responsive  and  letter 
strict  to  the  requirements  of  DOD  Directive  5000,1  and  the 
desires  expressed  by  Congress  in  Ihiblic  Lav/  92-156  Section  506, 
covering  their  desire  for  appropriate  operational  test  reports, 
the  initial  PF  test  plan  would  bo  unacceptable.  However,  if 
v/hole  ship  lOT&E  is  to  be  completed  and  results  published  and 
. analyzed  before  the  initial  production  decision  the  delay  in 
delivery  of  each  ship  could  be  as  much  as  three  and  a half 
years*  According  to  the  GNO,  the  requirements  of  the  Navy 
make  this  unacceptable.  A middle  ground  intr.oducing  consider- 
ably less  delay  might  be  more  appropriate.  In  other  weapon 
system  acquisition  programs,  DDT&fi  has  agreed  that  continuation 
of  production  at  one  source  before  the  major  production  decision 
is  an  appropriate  measure  to  reduce  pi-ogram  costs  and  addition- 
ally provide  units  on  which  to  conduct  further  OT&E,  'fhis  ■ 
I,)rccedent  indicates  the  OSD  criticism  would  be  directed  only 
toward  the  beginning  of  production  at  the  second  and  third 


shipyard  in  advance  of  the  results  of  lOT^E  from  the  lead 
ship. 


D,  DSARC  I RESULTS 

On  August  1972,  the  PF  Project  Manager  presented  the 

PF  p-’ogram  to  the  Defense  Systems  Acquisition  Reviev/  Council, 

The  follov/ing  is  Deputy  Secretary  of  Defense  Rush's  response 

to  DSAI^vC  I with  respect  to  Test  and  Evaluation: 

I am  pleased  to  note  the  strong  effort  to  insure  adequate 
test  and  evaluation  (including  lOT&E)  prior  to  major 
contract  for  follow  ships.  However > the  planned  date 
for  the  first  major  contract  for  follow  ships  assumes 
that  no  critical  deficiencies  will  be  found,  during  such 
testing.  The  Navy  should  continue  to  give  emphasis'  to 
the  completion  of  all  feasible  early  T&E  (including  lOTS^E) 
on  the  combat  subsystems  and  on  the  land~based  test  sites, 
•The  DSilRC  and  the  DDT&E  will  evaluate  at  the  time  of 
their  review  of  the  Navy's  recommendation  to  proceed  with 
follo'w  ships  whether  adequate  test  and  evaluation  (in- 
cluding I0T2<E)  has  been  accomplished  with  satisfactory 
results,  and  if  not,  whether  some  delay  in  contractiiig 
is  v/arranted. 

Also,  it  may  be  desirable  that  a period  for  onerational 
test  and  evaluation  of  the  lead  ship,  prior  to  that 
ship's  full  release  to  normal  Fleet  usage,  be  allocated' 
to  OPTEVIi’OR,  T.he  purpose  of  this  testing  would  be  to 
determine  the  PF's  expected  operational  effectiveness 
in  its  expected  roles  and  the  need  for  any  early  modifi- 
cation to'  follow  ships.  Should  such  modification  bo  re- 
quired, a later  DSM^'G  u'ould  have  to  determine  the  rela- 
tive merits  of  opening  existing  contracts  to  change  by 
change  order  procedures  or  making  modifications  after 
acceptance  from  the  shipbuilder(8: 1-2), 


E.  INITIAL  LAND  BASED  TESTING  RESULTS 


Aklthough  it  is  too  early  to  jjredict  the  final  results 
of  the  PF  T&E  program,  the  ability  of  the  LBTSs  to  generate 
initial  assossjnent  data  of  the  PFs  opera'kional  capabilities 
has  been  significant.  OT&E  and  development  personnel  have 
conducted  such  thorough  and  extensive  testing  at  the  LBTSs 


that  as  a result  of  DSARC  II,  SocDef  waived  the  requirement 
for  a DSAKC  III,  The  lead  ship  FFG-7  was  launched  on  25  Sept 
1976  and, expects  to  bo  delivered  to  the  Navy  in  Oct  1977* 
Fabrication  is  currently  scheduled  to  begin  on  the  follow-on 
ships  in  May  1977.  Although  the  Program  Management  Office 
is  very  optimistic,  the  real  proof  of  the  LBTSs  operational 
assessment  value  will  be  readily  evident  during  the  at-sea' 
operational  test  trials  scheduled  to  be  conducted  in  1978. 


SECTION  III 
AIHCRAFT  ACQUISITION 

la  conti'ast  to  the  low  risk,  long  fabrication  program  . 
of  conventional  hull  ships , the  aircraft  acquisition  is  one 
of  high  risk  and  uncertainty  with  a relatively  short  acquis- 
ition cycle.  In  mcst  aircraft  or  aircraft  related  v/eapon 
systems  programs  tl:.e  overall  test  program  is  very  complex 
and  is  subdivided  into  elements.  The  manner  in  which  the 
test  program  is  subdivided  depends  on  the  planned  test  organ- 
ization, location  of  facilities,  time  frame,  and  other  factors, 

A.  D'WELOPMKNT  TESTING 

Early  testing  in  the  exploratory  development  stage  is 
designed  to  investigate,  test,  or  evaluate  the  soundness  of 
a concept,  device  or  system  in  a breadboard  or  rough  exper- 
imental form,  without  regard  to  the  eventual  overall  design 
or  final  form.  The  advanced  development  stage  usually  involves 
a model  of  the  complete  system  or  integral  peurts  of  the  system 
for  experimentation  or  testing  to  demonstrate  the  technical 
feasibility  of  the  design  and  its  ability  to  meet  existing 
performance  requirements,  and  also  to  secure  engineering 
data  for  further  development.  The  final  advanced  develop- 
ment model  will  approach  the  required  form  factor.  Serious 
consideration  will  be  given  to  military  requiremeuts  such  as 
reliability,  maintainability,  human  factors,  and  environmen- 
tal conditions.  In  the  fiill  scale  development  stage, 


engineering  or  operational  tests  under  Se|rvice  conditions 
are  conducted  to  evaluate  performance  and  military  suit- 
ability. The  prototype  aircraft  system  v/ill  closely  ap- 
proximate aji  initial  production  design,  have  the  required 
form,  and  will  meet  the  standard  militeiry  requirements 
such  as  reliability,  maintainability,  human  factors,  extreme 
environmental  conditions,  etc.  ■ , 


B.  AIRCRAFT  TEST  PR0GRAJ4  REQUIREMENTS 

The  Navy  aircraft  test  program  requirements  are  pre- 
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sented  in  figure  4.  These  tests  take  place  in  the  full 
scale  development  phase  of  the  procurement  cycle.  The  pri- 
mary purposes  of  these  tests  are; 

1)  To  determine  that  the  aircraft  can  be  safely  oper- 
ated by  Navy  jdlots  during  in-flight  trials  to  limits  con- 
sistent with  the  contract  design  limits; 

2)  To  obtain  early  basic  information  regarding  the  mili 
tary  potential  of  nev;  models  of  aircraft  and  the  operability 
of  all  their  equipment; 

3)  To  permit  eai’ly  decisions  regarding  attainment  of 
superior  characteristics; 

4)  To  obtain  quantitative  information  on  safe  limits 
for  operation  by  fleet  pilots(  1 1 ;7l'> , 

After  the  first  flight,  but  prior  to  initial  delivery 
of  the  aircraft  for  trials,  Navy  test  pilots  designated  by 


KAVr  AIRCRAFT  TEST  PROGRAK  REQUIREMEKTS 


FIGURE  4 
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the  Commander,  Ha.val  Air  Test  Center , Patuxent  River,  Maryland, 
conduct  normally  in  five  phases,  the  Navy  Preliminary  Evalua- 
tion (NPE)«  A phase  consists  of  one  or  more  flights  by  the 
Evaluation  Team  and  each  phase  is  terminated  by  the  COMAVAIR- 
TESTCEN,  Phase  I of  the  NPE  is  performed  immediately  subse- 
quent to  the  contractor's  inspection,  normally  about  90  days 
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after  the  contractor's  first  flight  of  the  aircraft.  Addi- 
tional phases,  as  required,  are  performed  at  times  appropri- 
ately related  to  the  development  of  the  design  by  the  contractor 
as  the  allowable  flight  envelope  is  increased,  or  to  evaluate 
changes  incorporated  in  the  aircraft  to  correct  deficiencies. 

The  final  phase  is  scheduled  Just  prior  to  initial  delivery 
of  the  aircraft  for  Board  of  Inspection  and  Survey  (BIS)  Trials, 
The  NPE  flights  ai-e  made  at  the  contractor's  facility  unless 
otherwise  authoriaed  by  the  COMNAVAIHTESTGEN. 

The  purposes  of  the  Navy  Preliminary  Evaluation  are  as 
follows; 

1)  To  determine  at  the  eai-liest  possible  opportunity 
the  combat  potential  and  gross  deficiencies  of  the  aircraft 
and  thereby  enable  an  estimate  to  be  made  of  the  degree  to 
which  operational  requirements  will  be  met, 

2)  To  highlight  the  need  for  and  to  allov?  early  coriec- 
tion  of  deficiencies, 

3)  Evaluate  the  aircraft  weapons  system  installation 
including  functional  and  accuracy  chocks  of  gun,  bomb  si.ght. 
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rocket  rack,  etc*,  and  flight  tessts  of  fire-control  systems 
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and  firing  runs  at  a suitable  target, 

4)  Evaluate  critical  combinations  of  aircraft  weight 
and  C,G,  to  determine  aircraft  readiness  for  BIS  Trials ( 1 1 5 TO) • 
Follovdng  the  NPE  the  Board  of  Inspection  and  Survey 
Trials  are  commenced  to  determine  Service  suitability  and 
contractor  specification  conformance  with  production  aircraft. 
All  equipment  and  installations  specified  for  the  aircraft 
must  be  installed  'and  operable  except  for  aircraft  instrumented 
for  special  tests  in  which  weight  requirements  for  instruraont- 
ation  may  require  the  removal  of  certain  equipment.  In.  these 
special'  test  aircraft,  all  applicable  ai'mament,  electronic 
equipment,  and  other  items  that  influence'  aerodynamic  char- 
acteristics or  the  C.G,  positions  of  the  aircraft  must  be 
installed  o.v  simulated  appropriately  to  represent  aircraft 
scheduled  for  fleet  delivery.  All  discropencies  renorted 
from  the  NPE‘s  must  have  been  corrected  unless  otherwise 
authorized  by  NAVAIR8YS00M. 


0.  CU.RREHT  PROBLEmG  XW  AI.RCR.A.FT  TEST  AND  EVALUATION 

Some  of  the  significant  facton-i  that  consistently  distort 
the  orderly  flow  of  aircraft  test  and  evaluation  arc  discussed 
below. 

Concurrency 

The  Board  of  Inspection  and  Curvey  generally  receives 
the  production  aircraft  right  after  those  designated  for 
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contractor  test  and  demonstration  purpoejes.  The  next  sub- 
block-of  aircraft  produced  go-  to  COMOPTTiivFOR*  s aircraft  test- 
ing activities  (VX-1 , VX-4,or  VX-5),  The  next  block  of  air- 
craft produced  are  delivered  to  the  fleet  (usually  to  squadrons 
designated  to  train  the  initial  group  of  aircrewman  and  main- 
tenance personnel).  The  point  is  that  within  one  or  two  months 
of  BIS’s  receipt  of  its  aircraft*  the  aircraft  in  in  the  fleet. 
The  lead  time  OPTEVFOR  has  is  slightly  less.  This  degree  of 
concurrency  has  two  effects.  Because  the  BIS  aircraft  follow 
closely  on  the  heels  of  the  contractor's  test  article,  they 
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do  not  contain  the  fixes  th-at  have  turned  up  in  the  contractor's 
testing  program.  Thus  BIS  airplanes  are  not  representative 
of  the  operational  configuration,  nor  axe  the  OPTEVFOR  or 
early  fleet  dolivories. 

Secondly,  because  the  production  lino  is  running  while 
BIS  and  operational  testing  are  in  progress  the  problems  ident- 
ified in  Navy  T^.-F  are  gener-'-'lly  not  incorporated  in  oven  early 
deployed  aircr-aft,  S-a.fety  of  flight  and  other  extremely  cru- 
cial fixes  are  incorporated  by  factory  teams  in  early  opera- 
tional squadron  aircraft  just  prior  to  or  after  deployment. 
Hopefully  some  of  those  problems  v/ere  identified  in  the  NPK's, 
but  it  must  be  remembered  that  these  tests  are  conducted  upon 
aircraft  that  are  even  less  representative  of  the  operational 
configuration  than  the  BIB  or  operational  test  aircraft. 

To  preclude  e lengthy  delay  (1  to  2 years)  betv/een  pro- 
duction of  tlK?  contr-actor  and  Navy  test  articles,  and  the 
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operational  aircraft}  a considerable  degree  of  concurrency 
is  planned  into  the  program*  Then  as  schedule  slides  occur 
•in  earlier  program  activities,  the  concurrency  between  Navy 
T&E  and  production  is  increased  because  initial  operational 
dates  are  rerely  changed,  to  accommodate  test  and  evaluation. 
That  is,  an  unfortunate  initial  situation  tends  to  worsen 
during  the  program.  The  results  of  this  concurreiicy  are; 

1)  A large  number  of  Engineering  Change  Orders  (EGO’ s), 

2)  Ind.tial  satis  faction  of  the  fleet  with  many  new 

aircraft, 

3)  A large  number  of  different  configurations  of  a par- 
ticular aircraft  in  the  fleet  to  be  supported, 

h)  Doloys  in  positioning  of  spares  to  suppiort  systems 
modified  by  ECO’s  and  late  production  oh'-inges, 

Addi.tlovial  life-cycle  costa  due  to  rework  or  ccrapning 
and  reprocurement  of  existing  spares . 

6)  Degradation  of  training  because  of  inoper-able  or 
faulty  weapons  systems. 

Late  Test  Data/Re su Its 

Since  the  recGint  of  Navy  Tf-E  test  data  and  results  by 
the  procuring  activity  .are  generally  critical  and  to  late  to 
affect  the  original  pregram  technical  decisions,  they  have 
the  effect  of  seeming  to  rut  tbo.;  Systems  Command  personnel 
"on  report",  A defensive  attitude  on  the  part  of  the  Progiora 
Manager  and  the  Systems  Command  technical  branches  inevitably 
results.  The  cost  of  solving  and  implementing  the  solvitions 
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for  these  problems  is  high*  Thus  the  tejst  report  tends  to 
embarrass  the  program  team  and  provide  ammunition  for  program 
critics  both  within  and  without  the  Navy  who  have  their  ovm 
' axes  to  grind,  instead  of  being  a tool  for  the  acquisition 
and  fielding  of  the  best  weapons  system.  Unfavorable  test 
reports  are  usually  buried  or  at  least  not  acted  upon  with 
v/hole-hearted  vigor  and  determination.  If  a test  report  from 
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another  activity  can  be  found  that  disagrees  with  the  unfavor- 
able test  report  in  any  substantial,  or  occassional.ly  insub- 
stantial manner,  it  is  used  to  cast  doubt  on  the  validity  of 
the  unfavorable  report*  These  statements  are  not  intended 
to  besmirch  the  characters  of  systems  command  personnel  but  to 
simply  point  out  the  natural  human  reaction  to  a test  report 


received  too  late  to  aid  in  a decision  already  made  and  im- 
plemented which  soys,  in  effect,  that  the  wrong  decision  was 
made , 


Non-Renrosontative  Hardware 

^s  discussed  in  the  previous  section  on  concurrency, 
test  aircraft  and  early  operational  aircraft  configurations 
are  in  a continual  state  of  flux*  The  result  is  that  tech- 
nical (BIS)  and  operational  testing  may  be  conducted  on  sys- 
tems that  will  later  undergo  substantial  chonges.  Some  changes 
obviously  invalidate  prior  test  results;  other  changes  do 
also  but  it  is  not  at  all  obvious  in  advance.  The  com- 
munity is  frequently  called  unon  to  dotorraine  whether  to  delay 
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tests  av/aiting  a modification  they  know  vdll  take  place  or 
proceed  on  the  tentative  assumption  that  the  change  will  not 
invalidate  the  test  results.  Delaying  tests  is  rarely  chosen 
unless  invalidation  is  apparent.  Although  systems  integration 
testing  of  a change  is  commonly  done  by  the  contractor  on 
the  computer  in  the  laboratory  or  on  prototypes,  it  can  not 
be  depended  upon  to  reveal  second  order  or  higher  effects 
and  suffers  from  the  basic  lack  of  complete  reality  inherent 
in  simulations  and  the  laboratory. 

In  summary,  TOE  is  conducted  on  an  aircraft  that  differs 
markedly  from  the  operational  aircraft.  Some  of  these  dif- 
ferences are  incontiequential  but  many  are  not.  The  worst 
facet  of  this  situatioh  is  not  that  the  test  v/ork  done  may 
become  invalid,  but  that  adequate  TOE  yitl'i  not  be  conducted 
on  the  operational  configuration  aircraft.  This  .situation 
causes  the  fleet  to  be  the  one  v/ho  identifies  many  consequen- 
tial problems  and  leads  the  fleet  to  feel  with  some  justifi- 
cation that  it  is  building  its .own  'Ircraft, 

CpmTiotitlon  Among  Wavy  TitE  Activl  t1 '■s 

One  of  the  major  bars  to  efficient  and  economical  test- 
ing is  the  high  degree  of  comnetition  among  and  v/ithin  the 
testing  activities.  As  the  mil.itai'y  pie  has  gotten  smaller 
in  rea,l  terms,  nrossures  to  incronso  a p.-uaticular  activity’s 
slice  to  the  detriment  of  another  activity  have*  grown  much 
3.argC!r,  Brochurem.ansVLli-)  has  become  tlio  order  of  the  day,  A 
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perusal  of  the  mission  statements  of  several  ma;jor  aircraft . 
testing  activities  within  the  Navy  reveals  a large  amount  of 
overlap.  This  overlap  results  as  each  activity  attempts'  to 
enlarge  its  scope  and  potential  budget  via  its  mission  state- 
raont.  This  same  disease  exists  within  major  activities  as 
divisions  and  branches  vie  for  the  T&E  budget.  The  temptation 
here  is  for  each  activity  to  overestimate  its  expertise  and 
facilities  or  plan  on  developing  the  necessary  expertise  and/or 
facilities  after  receipt  of  a task.  This  procedure  is  extreme- 
ly wasteful  of  resources  because  it  leads  to  duplication  of 
facilities  and  expertise  and  the  conduct  of  testing  by  act- 
ivities not  particu3.arly  v/ell  qualified  to  do  so. 

While  Navy  Industrial  Funding  (NIF)  does  provide  an 
Incentive  to  efficiency  in  test  work  it  also  feeds  the  firo 
of  intense  and  destructive  competltj on.  As  in  tho  civilian 
v/orld,  tho  primary  goal  of  any  Navy  organi^.ation  is  survival, 
and  the  means  of  long-term  survival  is  to  expar.d  into  other 
testing  activities’  areas  of  competence,  A further  u)if ortun- 
ate  aspect  of  NIF  is  that  some  testing  activities  are  NI.F 
funded,  e,g.  the  Naval  Air  Test  Center  (NATO),  and  tho  Naval 
Weapons  Fvaluntion  Center  (NV.'KF)  and  others  ore  not,  such 
as  OP'I’FVFON,  so  that  testing  tends  to  go  where  costs  are 
lower  without  due  regard  to  comnot(?nce  a>ni  comxiar ability  of 
actual  coats  to  tho  Navy, 

The  result  of  this  intense  comp-ftition  is  that  comnet- 
ing  tooting  activitio.':',  find  it  very  difficult  to  cooT)orato 
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fully  on  a tna;5or  testing  program.  What  has  been  said,  v/ith  , 
regard  to  funding  is  equally  true  of  test  assets » particular*^ 
ly  test  aircraft,  • . 

An  attempt  to  ameliorate  this  competition  is  being  imple- 
mented nov/.  " An  existing  organization!  the  Naval  Air  Systems, 
Command  T&E  Coordinator  > has  been  empowered  to  designate  where 
aircraft  testing  will  take  place  and  by  v/hom  it  v;ill  be  done. 

Of  course , the  TSsE  Coordinator  will  be  faced  with  end  runs  to 
OPNAV  and  the  Program  Manager  by  diBsatisfjed  I'&E  activities, 

P,  PROTOTYPING 

As  a result  of  Deputy  Director  of  Defense  David  Packard’s 
ideas > prototyping  has  gained  a great  deal  of  attention. 

Another  name  for  total  system  prototyping  is  "fly  before  buy". 
Visions  of  competitive  fly-offs  and  the  lilse  arose.  The  Air 
Force  has  conducted  several  successful  competitive  fly-offs 
including  the  AX.,  but  total  system  prototyping  is  frequently 
not  an  economical  approach.  It  is  an  emotionally  attractive 
technique  in  that  it  vastly  reduces  the  degree  of  risk  in 
major  decisions,  but  the  increase  in  cost,  time,  and  re- 
sources is  great  and  may  not  always  be  available.  However 
this  interest  in  actual  hardware  signalled  a decrease  in 
dependence  on  paper  studies  which  existed  during  lir,  McNamara’s 
tenure  in  DOD,  It  is  likely  that  increased  sequential  com- 
ponent or  sub-system  proofing  at  lower  levels  of  development 
will  prove  to  be  the  most  beneficial  aspect  of  -this  change 
rather  than  the  few  "fly  before  buy"  or  competitive  fly-off 


sytstems  that  have  occurred. 

E,  INITIAL  g-lZ^  TICST  MW  EVALUATION  PLAN 

A more  economical  approach-  to  the  improvement  of  T&E 
v/^  planned  in  the  Navy's  F-lif  T&E  program.  A key  , factor 
in  this  approach  was  the  selection  of  the  NASC  T&E  Coordinator 
to  supervise  the  generhtion  of  integrated  tost  .pl^s,  the 
allocation  of  test  res ourc,o,s»  ,and  of  the  actual  testing  and 
witnessing.  I'he  testing  requirements  of  the  technical  test- 
ing activities,  the  Board  of  Inspection  and  Survey,  and  OP-' 
TEVFON  v;ere  integrated  into  a single  test  i^lan.  The  major.,ty 
of  the  Navy  developmental  and  operational  testing  took  place 
during  the,  saiiie  period  and  even  on  the  saine  flights.  Maximum, 
use  was  made  of,  contractor  deraonsti’a-bions.  witnessod  by  thO' 
Navy  testing  activities  to  obviate  t)ae  retesting  of  a tech- 

nical  point  already  demonstrated  by  the  contractor.  Witness- 

1 • 

ing  by  testing  activities  v/as-  crucially  important  and  allowed 
tne  contractor's  data  to  be  readily  accepted  by  the  testing 
activities.  This  approach  also  helped  to  eliminate  redundancy 
in  testing,  i.e*  the  testing  of  the  siame  performance  parameter 

5 ' • . ■ - 

\ . 

by  fijjd'-veral  different  activities  which  luis  been  a consistent 
and  wasteful  feature  Navy  testing  in  the  past. 

Obviously,  this  approach  places  a great  deal  of  respons- 
ibility directly  on  the  shoulders  of  the  TIE  Coordinator,  and 
requires  his  staff  \to  deal  knowledgeably  with  a ’wide-ranging 
and  complex  test  plan.  The  potential  for  major  improvements 


in  Naval  aircraft  testing  is  evident  but  it  depends  greatly 
on  the  competence  and  the  resources  of  the  T&E  Coordinator's 
staff  and  the  T&E  Coordinator's  ability  to  resist  strong 
pressures  from  various  special  interest  groups  in  the  T&E 
community  (OPTEVFOE,  NATO,  etc,).  The  intense  competitive 
pressures  previously  mentioned  still  exist  and  must  be.  re- 
sisted by  the  T&E  Coordinator  or  the  potential  of  this  ration 
al  approach  will  flounder  in  interactivity  politics. 
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SECTION  IV 


CONCLUSIONS  I j . 

It  is  readily  appai'ent  that  significant  differences 
exist  in  ship  and  aircraft  acquisition  cycles.  Whereas  a 
conventional  hull  ship  acquisition  program  is  one  of  low 
risk  and  long  fabrication!  an  aircraft  acquisition  is  one  of 
high  risk  and  uncertainty  v;ith  a relatively  short  acquisition, 
cycle.  Therefore  the  test  and  evaluation  requirements  estab- 
lished by  DOD  must  be  tailored  to  fit  each  program, 

DOD  Directive  5000* 1 states  that  a determination  of  oper- 

I 

ational  suitability  will  be  made  prior  to  large  scale  produc- 
tion commitments.  However,  it  does  not  appear  that  the  Navy 
intends  to  vmit  one  or  two  years  to  start  production ' of  a 
system  after  the  completion  . of  ojjerational  test  aj:'tic,les.  In 
the  PF  program,  the  delay  to  the  program  of  v/aiting  to  stai't 
construction  of  the  follow  ships  until  the  lead  ship  was  built 
and  operationally  tested  was  estiniatod  . to  bo  15  months.  Since 
this  delay  was  unacceptable,  the  decision  v;as  wade  to  st^’'t 
production  on  the  first  half  of  the  ships  prior  to  the  comple- 
tion of  the  Initial  Operational  Test  and  Evaluation,  However, 
the  production  on  the  second  half  would  be  deferred  until  I0TS.-S 
v/as  complete.  It  seems  unlikely  that  extended  delays  to  com- 
plete lOT&E  will  ever  be  accepted  in'  a major  system  acquisition. 
The  trend  since  19?2  has  been  to  allow  OPTSVFOR  an  earlier  start 
on  their  operational  evaluation  and  a better  power  position 
with  respect  to  the  technical  testing  community.  But,  no  great: 


improvements  in  T&E  are  expected  until  the  basic  human,  con- 
currency, and  hardv;are  problems  which  reduce  T&E  effectiveness 
have  been  changed. 

The  systems  approach  to  testing  may  be  the  best  answer 
to  the  problems  associated  with  Navy  T&E,  Little  faith  is 

placed  on  policies  that  simply  shift  the  T&E  power  structure 

/ ’ 

(OT&S)  or  pretend  that  concurrency  can,  be  eliminated  (proto- 
typing), The  P-lZf  testing  concept  appeared  to  be  a step  in, 
the  right  direction however  funding  problems  developed  and 
a significant  portion  of  the  test  program  was  out,  out,  Since 
the  United  States  will  always  be  reacting  to  newly  perceived 
existing  or  future  threats,  there  will  never  be  enough  time 
to  develop  major  weapons  systems  without  concurrency  in  the 
development,  test  and  evaluation,  and  production  phases  of 
the  acquisition.  Leisurely  serial  development,  test,  and 
production  of  major  v/eapons  systems  is  an  unrealistic  pro- 
cedure to  hope  or  plan  for.  The  only  realistic  hope  for 
decreasing  the  concurrency  of  test  and  evaluation  and  pro- 
duction is  to  reduce  the  length  of  the  Navy  T&E  process  through 
improved  test  planning,  improved  efficiency,  and  the  reduction 
of  redundancy  in  testing.  It  appears  that  coordinated  and  in- 
tegrated testing  is  the  most  likely  approach  to  result  in 
significant  improvement  in  Navy  T&E  and  bears  investigation 
by  other  major  acquisition  programs  for  ai)pliGability  to  their 
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